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The problem of this' study was to determine the effect 
of an inservice education program of evaluation on school-based 
personnel in Escambia County, Florida, as measured by cognitive 
testing (knowledge) and affective (attitudinal) instrumentation. In 
addition, the determination of the specific objectives, met, or not 
met, by inservice participants was an important component of this 
study. (Author/CK) 
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A STUDY OF AN INSERVICE PROGRAM 
OF EVALUATION IN ESCAMBIA COUNTY, 
FLORIDA PUBLIC SCHOOLS 



The Problem 



The problem of this study was to determine the effect 
of an inservice education program of evaluation on school- 
based personnel in Escambia County, Florida as measured 
by cognitive testing (knowledge) and affective (attitu- 
dinal) instrumentation. In addition, the determination 
of the specific objectives met, or not met, by inservice 
participants was an important component of this study. 



Procedures 



The population of the study was twenty-four 
elementary principals and curriculum coordinators of ^ 
target Title I schools in and around Pensacola, Florida. . 
Analysis of the data collected through cognitive, affective 
testing, and surveys, made it possible to numerically re- 
present the effect of the inservice program based on the 
measurement instrumentation employed. 

The instruction amounted to fifty-four hours , 
presented over twelve meetings . The instructors were 
university consultants, or presentors from within the 
school district. The main topics covered were, federal 
requirements, evaluation and measurement, needs assess- 
ment, objectives data processing, sampling and design, and 
statistics. A control group of non-participants was 
selected for comparison purposes • The main instrumentation 
used was a cognitive test constructed from instructional 
objectives. Population variables such as educational 
experience, interest in topics presented, age, and educa- 
tional position were presented in an intercorrelation ana- 
lysis with the cognitive, and affective instrumentation. 



Findings and Conclusions 



A significant difference at the .01 level of proba- 
bility was found in favor of the inservice participants 
over the control group on the cognitive test measure. 
Statistical significance was not found between the attitude 
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measure of the control group, and the same measure for the 
inservice participant post-test. Also, no significant 
change in attitude, as measured by the instrument used, 
resulted from pre-test to post-test for the inservice group. 

Attendance was found to be an indicator of possible 
success within the inservice group on the cognitive test. 

The intercorrelation analysis of population variables for 
the inservice group did not yield significant correlations 
with the exception of one significant negative correlation. 
The significant negative correlation found was between 
research related college course background, and the cognitive 
test results presented in this study. 

The cognitive test items and correct answers for the 
test used in this study are included in this paper. Finally, 
a list of objectives met, or not met, for the inservice 
group of administrators and supervisors is presented. 
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RANKING OF INSERVICE PARTICIPANTS BY COGNITIVE 
TEST SCORES SHOWING ATTENDANCE, ATTITUDE, 

INTEREST SCORES, EDUCATIONAL EXPERIENCE, AND COLLEGE TRAINING 
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participants will be able to 5 . workshop participants will be 
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INSERVICE PARTICIPANT GENERAL OBJECTIVES BY 
SESSION AND SPECIFIC OBJECTIVES SHOWN BY SESSION 
THAT WERE MET OR NOT MET* 



participants will learn the 11. participants will identify one 

results of The 1969, Florida result of the 1969, Florida State 

Statewide Needs Assessment Needs Assessment 
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GENERAL OBJECTIVES SPECIFIC OBJECTIVES HET MET 



INSERVICE PARTICIPANT ITEM DISTRIBUTION 
BY PERCENTAGE OF ITEMS CORRECT 
ON THE COGNITIVE TEST 
N = 24 



0 - 49% 
Correct 
Test Item No. 



50 - 62% 
Correct 
Test Item No. 



63 - 75% 
Correct 
Test Item No. 



76 - 100% 
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Test Item No. 
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CONTROL GROUP 

PERCENTAGE OF ITEMS CORRECT 
PLOTTED ON THE INSERVICE 
PARTICIPANT DISTRIBUTION 
N = 27 



0 - 49% 
Correct 
Test 
Item No. 
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Test 
Item No. 



63 - 75% 
Correct 
Test 
Item No. 
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Cog niti ve Test Questions 

Which of the following groups are not identified specifically 
as eligible for Title I Programs: 

(a) handicapped 

(b) delinquents 

(c) pre-school children 

(d) adults 

(e) dropouts 

One difference between evaluation and research is in the 
use of : 

(a) statistical analysis 

(b) value judgements 

(c) computers 

(d) data collection 

If accountability is taken as an ''end," evaluation can be 
considered to be the: 

(a) same 

(b) means 

(c) opposite 

(d) beginning 

As a concept a evaluation can be best equated with: 

(a) process 

(b) procedure 

(c) data 

(d) decisions 

Evaluations should be planned by : 

(a) a systematic approach 

(b) the project director 

(c) children 

(d) a standardized test 

Potential problems found in evaluation include: 

(a) the use of invalid testing procedures 

(b) lack of local norms 

(c) lack of time 

(d) all of these 

All of the following are acceptable measurement devices 
for use in evaluations except : 

(a) norm- referenced testing 

(b) criterion -referenced testing i . 

(c) testimonial letters 

(d) attendance check lists 
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8. In criterion referenced testing* it is important to know: 

(a) whether norms are met 

(b) which objectives are met 

(c) which objectives are not met 

(d) both b and c 

9. One way in which a norm reference test differs from a 
criterion referenced test is that the criterion referenced 
test has : 



(a) little value 

(b) no comparison group 

(c) no validity 

(d) no reliability 

10. The educational priorities in Escambia County School 
System's Title I Program are basically determined from: 

(a) accreditation standards 

(b) Advisory Council recommendations 

(c) needs assessment 

(d) School Board decisions 

11. One result of the Heeds Assessment conducted by the 
State has been to: 

(a) revalidate the 9th grade testing program 

(b) revise accreditation standards 

(c) encourage educational assessment programs 

(d) establish a new department in DOE 

12. The priority need of disadvantaged students as determined 
by the Escambia County weeds Assessment is : 

(a) teacher aides 

(b) reading 

(c) summer programs 

(d) free lunch 

13. Title I Programs must agree completely with: 



(a) accreditation standards 

(b) Title I Advisory Committee Policy 

(c) Title I Guidelines 

(d) all of these 

14. Often, a list of program objectives will include multiple 
domains such as : 




(a) cognitive 

(b) affective 

(c) cognitive and affective 
( d ) ps ychomotor 

17 
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15. The next logical statement to follow the general objective 
is the : 

(a) behavioral criterion 

(b) specific objective 

(c) experimental design 

(d) needs statement 

16. In the final analysis, objectives may best be described 
as statements of: 

(a) intent 

(b) attitudes 

(c) fact 

(d) both a and b 

17. Poorly written specific objectives often contain the 
following words : 

(a) appreciate , understand 

(b) cite , list 

(c) identify, differentiate 

(d) compare, contrast 

18. The process of writing objectives assists the program 
planner because: 

(a) it provides budget information 

(b) provides behavioral criteria 

(c) relates objective and evaluation 

(d) all of these 

19. An information system provides facts for; 

(a) decision making only 

(b) effective administration only 

(c) research only 

(d) curriculum development only 

(e) all of these 

20. An important component of a data system is: 



(a) personnel 

(b) procedures 

(c) coding capability 

(d) none of these 

(e) all of these 



21. SIRS would provide an integrated data base for: 

(a) student information 

(b) payroll information 

(c) systems information 

(d) none of these 
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22. The Escambia County Data Board was established to meet 
the data processing needs of the: 

(a) School Board 

(b) City of Pensacola 

(c) County 

(d) all of these 

23. Initial assessment procedures are designed to provide 
information about: 

(a) individual progress 

(b) instructional methods 

(c) accreditation levels 

(d) group characteristics 

24. Sampling techniques assume that: 

(a) parameters are known 

(b) a population is precisely defined 

(c) a population is normally distributed 

(d) none of these 

25. Internal validity is related to: 

(a) differences in pre-post scores 

(b) a level of .732 or better 

(c) a type of experimental design 

(d) the effectiveness of the design 

26. In an educational experiment, the history means: 

(a) pupil folders 

(b) unplanned events between pre and post tests 

(c) the rationale and hypothesis of the experiment 

(d) all of these 

27. "To install in children a life-long love of poetry” is 
an example of a: 

(a) cognitive objective 

(b) specific objective 

( c ) goal statement 

(d) . measurable objective 

28. An interest inventory is an example of: 

(a) subjective measu rement 

(b) criterion referenced measurement 

(c) unsatisfactory evaluation 

(d) norm referenced measurement 
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Experimental designs are used to: 

(a) evaluate effectiveness of a treatment 

(b) measure differences between groups 

(c) justify expenditures 

(d) set of educational programs 

An educational experiment that can be generalized is said 
to have : 

(a) good design 

(b) logical choice of subject 

(c) appropriate statistical analysis 

(d) external validity 

In drawing a sample, it is best to use: 

(a) randomi zation 

(b) alphabetical order 

(c) teacher recommendation 

(d) volunteers 

When constructing a test, factors to be considered are: 

(a) validity 

(b) reliability 

(c) usability 

(d) all of these 

Evaluation procedures are designed to measure: 

(a) group characteristics 

(b) internal validity 

(c) individual characteristics 

(d) efficiency of teachers 

Reliability is defined as : 

(a) content validity 

(b) external validity 

(c) consistency of scores 

(d) appropriateness of level 

The term "normal distribution" refers to: 

(a) a special statistic 

(b) the way grades are assigned 

(c) the shape of a frequency distribution 

(d) an ogive 

(e) none of these 
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"Mean” is defined as : 



(a) the middle score 

(b) the average score 

(c) the most frequent score 

(d) all of these 

(e) none of these 

The symbol 2 means : 

(a) product 

(b) sum 

(c) sigma 

(d) significance 

(e) none of these 

The standard deviation is a measure of the: 

(a) relationship of the mean and the median 

(b) validity of a test 

(c) variability of scores 

(d) reliability of a score 

(e) none of these 

The first step in analyzing data is to: 

(a) find the mean 

(b) rank the scores 

(c) find the range 

(d) tally 

(e) none of these 

Correlation is: 

(a) the common mean of two tests 

(b) the relationship between two sets of scores 

(c) technique for determining internal validity 

(d) a measure of external significance 

(e) none of these 

A frequency polygon is : 

(a) a method of analyzing data 

(b) the same as an ogive t 

(c) a shortcut in calculating the standard deviation 

(d) a graphed frequency distribution 

(e) none of these 

A correlation may be any value between: 



(a) 


0 and «> 




(b) 


+1.0 


to 


-1.0 


Cc) 


+100 


to 


: 

H 

o 

o 


(d) 


+1.0 


to 


0 


(e) 


none 


of 


these 
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43. The formula £ x is used to calculate the: 

N 

( a > mean 

(b) median 

(c) standard deviation 

(d) 50th percentile 

(e) none of these 

44. In figure A, the mean falls in the interval: 

(a) 4-6 

(b) 7 - 9 

(c) 10 - 12 

(d) none of these 

45. In figure A, the median falls in the interval: 

(a) 4 - 6 

(b) 7-9 

(c) 10 - 12 

(d) none of these 

46. Figure A is an example of: 

(a) a normal distribution 

(b) a frequency polygon 

(c) a histogram 

(d) none of these 

47. Figure B is an example of: 

(a) a normal distribution 

(b) an ogive 

(c) a platykurtic distribution 

(d) none of these 

48. A t-test is one measure of: 

(a) statistical regression 

(b) statistical significance 

(c) reading vocabulary ~ 

(d) test validity 

49. Confidence limits in statistics represent the: 

(a) theoretical foundation of test construction 

(b) technique of item analysis 

(c) probability that results are due to chance or error 

(d) desired outcome of the program 
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. of Evaluation Services is to provide 
data^^pupil^petfo^ 06 

(a) administrative decisions 

(o) instructional 0 decisions 

(d) learner decision 

~ i:L ° ~ " . st nearly like which of the 

The standard deviation is most nearly 

following: 

( a ) dress 

(b) shoe 

(c) jewelry 

(d) girdle provided by Evaluation 

An of the following services a P 

and Testing Services except. 

• Of answer sheets for teacher-made tests 

ft ^Hnf ^analy- # ^ „ 

s xsMS34&‘ 

(e) interac Analysis Program 

The Test Analysis Progra... -hd Re3 P on 
can be used for. 

(a) standardized test t#gt8 

(b) teacher construe 

(c) questionnaires 

(d) survey 

(e) all of the se 
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The initial step in planning a program is 

(a , consultation with administrative staff 

(b) library research 

( c ) testing 

«> needs^ssessmenc dimens ions of 

The EPIC model for evaluation has the 
(a) content, ^ ^ /and 6 psy chomo tor 

SI 
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56. A behavioral objective consists of: 

(a) a student part and a teacher part 

(b) outcome and measurement , 

( c ) performer , expected outcome and performance standard 

( d ) action ana oenavior 

57. The Hawthorne effect can be: 

(a) helpful to the experimenter 

(b) both positive and negative 

(c) eliminated statistically 

(d) none of these 

58. Experimental mortality is a factor in: 

(a) external validity 

(b) pupil selection 

( c) internal validity 

(d) sampling procedures 

59. Program objectives are most nearly equivalent to: 

(a) behavioral criteria 

(b) needs statements 

( c) general objectives 

(d) specific objectives 

60 . XO is the symbolic representation for an experimental 
design known as : 

(a) time series 

(b) static group comparison 

(c) one shot case study 

(d) Solomon Four Group Design 

61. ROXO indicates that subjects were selected: 

(a) from remedial reading classes 

(b) alphabetically 

(c) according to the Rudovich Design 

(d) randomly 

62. Behavioral criteria are most nearly equivalent to: 

(a) specific objectives 

(b) evaluative instruments t , 

( c) techniques for behavior modification 

(d) performance standards 
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63. Objectives in the affective domain are: 

(a) difficult to measure 

(b) rarely written 

(c) similar to psychomotor objectives 

(d) both a and b 

64. Semantic differential is: 

(a) an experimental approach to workattack skills 

(b) a technique utilizing bipolar adjectives 

(c) suitable for pre-school children 

(d) none of these 



EXAMPLE A 



A third grade teacher decided to experiment with a new 
approach to reading instruction. She divided her class into two 
parts and used her regular method with one half and the new 
method with the other half. At the end of the year, she compared 
the reading levels of the two groups and found that the experi- 
mental group scored three points higher than the regular group. 



65. 


The 


experimental design for example A is: 




(a) 


excellent 






(b) 


good 






(c) 


fair 






(d) 


poor 




♦ 

CO 

CO 


The 


design in example A can be 


classified as 




(a) 


time series 






(b) 


pre-post observation with 


control group 




(c) 


static group comparison 






(d) 


none of these 





67. A possible source of internal invalidity in example A is: 

(a) selection y 

(b) experimental mortality 

(c) instrumentation 

(d) all of these 

(e) none of these 
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EXAMPLE B 



An inner city school developed a new program for disad- 
vantaged first graders . Children entering first grade without 
kindergarten experience were randomly assigned to standard and 
experimental first grade classrooms. At the end of the year* 
an analysis of test results indicated that children m the ex- 
perimental classrooms achieved at a level 9% higher than the 
average of the past five years. 

68. The design in example B can be classified as: 

(a) static group comparison 

(b) random pre-post observation 

(c) time series 

(d) observation with control group 

69. A flaw in this design tests with: 

(a) selection of comparison groups 

(b) assignment of pupils 

(c) choice of tests given 

(d) age of pupils 
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SUMMARY AND • RECOMMENDATIONS 



When considering the question of the effect of an 
evaluation inservice training program, one needs to examine 
a present educational trend. One of the most recent issues 
rflissd by school publics and educators has been accounta- 
bility. The author of this study believes that there is 
a certain amount of cognitive knowledge necessary for car— 
rying out the evaluation process so closely related to 
accountability. Further, an assumption is made by the 
author that the cognitive test utilized in this study has 
content relevant to the skills educators need to carry out 
responsible evaluation procedures. Evidence in this study 
indicates no significant relationship between the educational 
position of personnel in Escambia County and their results 
on the cognitive test for this study. Perhaps this suggests 
that no one group of educators should be the target group 
for carrying out the evaluation process . Instead the 
evaluation process might be just as easily approached by 
extending the knowledge necessary, and the responsibility 
involved, to all professional components of a school dis- 
trict. In addition, the cognitive test results in this 
study indicate that the subject matter content may be dif- 
ficult to transfer to practicing educators. 

Negative relationships between research related col- 
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lege courses and the cognitive results for this study may 
indicate that higher education does not now provide the 
necessary cognitive skills for carrying out an evaluation 
in a practical public school situation. At the least, it 
perhaps indicates that the subject content of the college 
courses may not be related to the content of the cognitive 
test used in this study. The fact that twenty out of 
twenty-four inservice participants did not score 70 percent 
on the test is not necessarily an indictment of the parti- 
cipants, for the situational and instructional process also 
play an important role. The cognitive test was difficult 
(63 percent), but considering the way the test items were 
constructed and related to content, the test may be consi- 
dered as having content validity for the cognitive instruc- 
tion. 

The length of sessions , amount of time between ses- 
sions , and the ability of the consultants and presentors 
to convey what they have to teach are of course important 
variables not specifically controlled for in this study. 
Future inservice programs might do well to consider the 
aforementioned three factors . The planning time allotted 
between the inservice planners , and the consultant or pre- 
senter, should be used wisely. The school district might 
fare better by providing extra care when setting objectives 




28 



- 28 - 



and contracting with consultants for exactly what is deter- 
mined as the inservice training needs of the respective dis- 
trict . 

Evaluating inservice programs in terms of objectives 
is a logical path for educators to follow. It is the 
author's opinion that specific objectives should be written 
or rewritten after each session has been completed so that 
the measurement devised is focused on what happened, not on 
what was projected to happen. Test questions and specific 
objectives might be better representations of content, and 
of better measurement quality, if the consultant or presen tor 
actually participated in their formulation. 

Considering the difference between the inservice group 
and control groups stratified with principals, central of- 
fice staff, and teachers, the cognitive success as measured 
by the test given in this study is a success indicator for 
the inservice program planners in Escambia County. 

In the future , attention might be given to the personal 
individual characteristics present in a group as diverse as 
the group that the inservice program was planned for in this 
study. Subject matter field diversity, individual interests, 
and especially the problems encountered daily in educational 
experiences, all are relevant targets for personalizing an 
inservice program to achieve better results. An Interest 
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Inventory administered in this study collected data about 
related areas of recent training in evaluation related to 
projected topics of the planned program. In terms of 
recency, the inservice participants responding did not in- 
dicate experiential strengths as a group in such areas as 
evaluation design, use of objectives in evaluation, data 
processing, statistics and the other areas listed. The 
preceding cognitive test results and the tables which list 
specific objectives met or not met, tend to indicate a suc- 
cessful overall effort by Escambia County Schools for en- 
gaging in the inservice program evaluated in this study. 

Using the criteria of below 50 percent (-1SD) as a 
group as the point at which an objective was not met, the 
specific objectives for the inservice program have already 
been presented in this paper in their entirety. In a case 
where more than one test question was criterion-referenced 
to only one objective, the number of items correct, and the 
place on the distribution, or standard deviation, were taken 
into account for making the judgment met or not met. 
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